7 research outputs found
Multi-Level Branched Regularization for Federated Learning
A critical challenge of federated learning is data heterogeneity and
imbalance across clients, which leads to inconsistency between local networks
and unstable convergence of global models. To alleviate the limitations, we
propose a novel architectural regularization technique that constructs multiple
auxiliary branches in each local model by grafting local and global subnetworks
at several different levels and that learns the representations of the main
pathway in the local model congruent to the auxiliary hybrid pathways via
online knowledge distillation. The proposed technique is effective to robustify
the global model even in the non-iid setting and is applicable to various
federated learning frameworks conveniently without incurring extra
communication costs. We perform comprehensive empirical studies and demonstrate
remarkable performance gains in terms of accuracy and efficiency compared to
existing methods. The source code is available at our project page.Comment: ICML 202
Learning to Optimize Domain Specific Normalization for Domain Generalization
We propose a simple but effective multi-source domain generalization
technique based on deep neural networks by incorporating optimized
normalization layers that are specific to individual domains. Our approach
employs multiple normalization methods while learning separate affine
parameters per domain. For each domain, the activations are normalized by a
weighted average of multiple normalization statistics. The normalization
statistics are kept track of separately for each normalization type if
necessary. Specifically, we employ batch and instance normalizations in our
implementation to identify the best combination of these two normalization
methods in each domain. The optimized normalization layers are effective to
enhance the generalizability of the learned model. We demonstrate the
state-of-the-art accuracy of our algorithm in the standard domain
generalization benchmarks, as well as viability to further tasks such as
multi-source domain adaptation and domain generalization in the presence of
label noise